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Abstract — Multimodal Biometric System using multiple 
source of information for establishing the identity has been 
widely recognized. But the computational models for 
multimodal biometrics recognition have only recently received 
attention. In the proposed system multimodal biometric images 
such as fingerprint, face, lung needle cancer images are 
extracted individually and are fused together using a sparse 
fusion mechanism. A multimodal sparse representation method 
is proposed, which interprets the test data sparse linear 
combination of training data, while constraining the 
observations from different modalities of the test subject to 
share their sparse representations. The images are 
preprocessed for feature extraction Based on the results 
obtained, label edge detection was used for feature extraction. 
Extracted features were subjected to sparse representation for 
the fusion of different modalities. 

The fused template can be used for separate the 
chromosome level for lung needle biopsy images. Lung needle 
biopsy image classification is a critical task for computer-aided 
lung cancer diagnosis. In this study, a Novel method, 
multimodal sparse representation-based classification (MSRC), 
is proposed for classifying lung needle biopsy images. 

In the data acquisition procedure of our method, the cell 
nuclei are automatically segmented from the images captured 
by needle Biopsy specimens. Then, features of three modalities 
(shape, color, and texture) are extracted from the segmented 
cell nuclei. After this procedure, MSRC goes through a 
training phase and a testing phase. These cell nuclei regions 
can be divided into five classes: four cancerous classes 
(corresponding to four types of lung cancer) plus one normal 
class (no cancer). The results demonstrate that the multimodal 
information is important for lung needle biopsy image 
classification. 

Index Terms — Feature Extraction, Dictionary learning, 
Lung needle cancer image classification, Multimodal 
biometrics, Sparse representation based classification (SRC). 


I. INTRODUCTION 

Unimodal Biometrics are systems that are capable of using 
only one physiological or behavioral characteristic for 
enrollment, verification or in reality, but the matching score 
is higher than the threshold, then he is treated as genuine. 
The proposed the seminal sparse representation based 
classification (SRC) algorithm for genetic algorithm [3]. It 
shows that by exploiting the inherent of data, one can obtain 
improved performance over traditional methods [1]. Lung 
cancer studied through by doctors but its grading gives 
different decisions which may differ from one doctor to 
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another. The classification and accurate determination of 
lung cancer grade is very vital because it influences and 
specifies patient’s treatment scheduling and finally their life. 
A new technique multimodal sparse representation-based 
classification (MSRC), suggested for classifying lung needle 
biopsy images [1]. In this technique data acquisition is done 
through the new method, the cell nuclei are mechanically 
segmented by itself from the input images caught by needle 
biopsy specimens which are obtainable in this research 
work, which is samples of the human thinking methods and 
the classification results are compared with some other 
computer-aided lung cancers analysis methods showing the 
efficiency of the proposed methodology. 

The three features modalities such as texture, color 
and shape are extracted from the segmented cell nuclei from 
input images. The training level, three discriminative sub 
dictionaries corresponding to three features information are 
together knowledgeable by a genetic algorithm directed 
multimodal dictionary learning approach. [8] The dictionary 
learning is used to select highest discriminative samples and 
encourage large disagreement amongst dissimilar sub 
dictionaries. In testing phase, when a novel image comes, a 
hierarchical fusion scheme is applied, which originally 
prediction of labels of all cell nuclei by fusing modalities 
such as color, texture and shape are predicts the label of the 
image by maximum popular voting. The above cell nuclei 
are as can be separated into five classes which consist of 
four cancerous classes and one regular class (non-cancer). 
Usually, lung cancer can be categorized into four types: 
squamous carcinoma (SC), adenocarcinoma (AC), small cell 
cancer (SCC), and nuclearatypia (NA). Fig. 1 shows several 
sample images of each of the four cancerous types. The 
results prove that the multimodal information is vital for 
lung needle biopsy image classification; this technique is 
particularly for classifying various cancerous types. The 
outputs of the proposed novel MSRC model presents 
reasonably higher accuracy, and are more similar with other 
existing algorithms [1]. 


Figure. 1 



NC SC AC SCC SA 


Fig.l: Typical examples of the normal (NC) images and 
four types of cancerous ones: SC, AC, SCC, and NA. The 
images are captured from needle biopsies Specimens by 
electronic microscopy and digital camera. 
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Traditionally, the diagnosis of lung cancer is made 
by the pathologist. However, the pathologist’s in experience 
in clinical practice can cause misdiagnosis. What is worse, 
the diagnosing process is time consuming and sometimes 
tedious when a pathologist is asked to analyze a huge 
volume of sample images from patients. Thus, the 
misdiagnosis can also be caused by the factors such as 
fatigue, even for an experienced pathologist. Therefore, the 
accuracy of the diagnosis is related to not only the 
pathologist’s educational background and clinical 
experience but also his/her physical and psychological 
conditions. In recent years, several computer-aided methods 
for lung cancer diagnosis have been developed, such as 
magnetic resonance imaging [2], computerized tomography 
[3] — [5], and X-ray chest films [6], etc. In clinical practice, 
analyzing the images captured from needle biopsy 
specimens, which routinely follows the X-ray chest films 
checking, is one of the most popular and reliable ways to aid 
lung cancer diagnosis. However, classifying different types 
(SC, AC, SCC, NA) of lung needle biopsy images is still a 
significant and challenging problem because the images of 
different cancerous types sometimes might be very similar 
to each other [2] and, hence, are difficult to classify. To 
tame the aforementioned challenges, we propose a novel 
method, named multimodal sparse representation-based 
classification (MSRC), for lung needle biopsy image 
classification. There are two Phase in Multimodal sparse 
representation. In the training phase, I will show the 
classification of lung needle cancer biopsy images and that 
converted into data acquisition methods and it gives the 
output to three feature extraction method [4]. it will perform 
the new sub -dictionary [4]. 


Fig.2: Framework of the proposed MSRC, including a 
training phase, the training phase outputs the 
discriminative sub dictionaries via genetic algorithm- 
based multimodal dictionary learning 

It aims to improve the classification performance, 
especially in the case that the images of different lung 
cancer types are difficult to distinguish. Our method exploits 
the observation that the lung cell nuclei have different 
appearances in different classes. Particularly, we investigate 
the information of three cell modalities (shape, color, and 
texture). For example, the SC cell nuclei are usually round 
(shape modality), and the NA cell nuclei always largely 
deviate from others in color (color modality). Upon the 
multimodal information, our method builds a Sparse 
representation-based classifier for lung needle biopsy image 
classification with improved performance. Next phase is 
testing phase, in this phase there are two level fusion 
describe [3] 

1) Cell level fusion 

2) Image level fusion 



Fig.3: Framework of the proposed MSRC, including a 
testing phase, the testing phase outputs the predicted 
label for each testing image. 

The rest of the paper is organized as follows. We 
first present the framework of our method and data 
acquisition procedure. Then, we introduce the feature 
extraction, training and testing phase, respectively. Finally, 
we present our experimental results and conclude the paper. 

II. LITERATURE REVIEW 

For lung needle biopsy image analysis, many 
efforts have been contributed in recent years. Zhou et al. [9] 
developed a lung cancer cell classification system based on 
a two-level neural network ensemble. In the system, the 
first-level ensemble determines whether a testing cell is 
normal or cancerous, and the second-level ensemble 
classifies the types of lung cancer for the suspected 
cancerous cells determined in the first level. Zhu et.al [10] 
proposed an image-level approach: multiclass multiinstance 
AdaBoost (MCMI-AdaBoost), which predicts the label of an 
image by incorporating the multi -instance distance 
measurement (Hausdorff distance) under the AdaBoost [11] 
framework. However, the similarity measurement ignores 
local information of cells. Shi et al. [8] recently introduced a 
transductive cost sensitive learning method for lung needle 
biopsy image classification, of which the goal is to achieve 
the best possible results with only a small number of labeled 
images. Unfortunately, the result of classification for 
different types of lung cancer is still unreliable, and there is 
still space for us to improve the classification performance. 
In addition, previous methods for lung needle biopsy image 
classification [9], [8] are single-modal based learning 
methods, which fail to make full use of the disagreement 
information among different modalities [8], [9]. It is 
noteworthy that, except for the needle biopsy specimens - 
based lung cancer diagnosis, many works that focus on 
cell/nodule/image classification in medical image analysis 
are highly related to our work. Works [2] belong to this 
category. For learning with multimodal data, also referred to 
multiview learning and ensemble learning in machine 
learning community, many algorithms [1] have been 
developed recently. Also, multimodal-based methods are 
promising in the field of medical image analysis since 
multimodal information is naturally available in the data 
acquisition procedures of various clinical tasks, such as 
Alzheimer’s disease diagnosis [2], [3], 

Prostate cancer prediction [6], and survival prediction for 
lung cancer [5]. 

The lungs are a complex tissue which contains 
numerous structures, such as vessels, splits, bronchi or 


Training Phase 
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pleura that can be situated near to lung nodules. Very 
Simple threshold approaches are regularly enough for the 
separation of solid well-circumscribed nodules, while lung 
nodules close to vessels or pleura need additional complex 
schemes exploiting geometrical and gray level features 
mined from nearby structures of the lungs. Various methods 
have been suggested to summary of the lung nodules close 
to vessels or pleura. Human associated parasites create a 
problematic in maximum humid countries, producing death 
or physical and psychological illnesses. Their conclusion 
regularly trusts on the visual examination of microscopy 
images, with error rates that may collection from moderate 
to higher. [3] The problem has been addressed through 
computational image analysis, but only for a rare species 
and images free of fecal layers. In routine, fecal layers are a 
actual trial for programmed image analysis. First method 
exploits ellipse matching and image foresting transform for 
image subdivision, multiple item descriptors and their 
optimal grouping by genetic software design for object 
symbol, and the optimum path forest classifier for object 
acknowledgment. The output demonstrations that this 
method is a favorable method near the completely 
automation of the entree parasitizes diagnosis. 

The effectiveness of sparse representations gained 
by learning a set of over complete basis or dictionary in the 
context of action recognition in videos. While the work 
focuses on distinguishing the movement of human, [6] 
physical appointments as well as the appearance of face and 
the suggested method is fairly general and can be used to 
address some other classification methods. The suggested 
approach is computationally effective, highly accurate, and 
is strong against partial sealing, spatiotemporal scale 
variants, and to some range to view changes. This 
robustness is attained by manipulating the discriminative 
nature of the sparse representations joined with spatio- 
temporal motion descriptors. The fact that the descriptors 
are mined over multiple temporal and spatial determinations 
makes them indifferent to scale variations. The descriptors 
being calculated locally make them robust beside blocking 
or other distortions. Features such as compressed sample can 
also develop the recognition accurateness but are highly 
costly computationally [8]. 

The presentation of the collective system is 
calculated in fact and the output correctness, rapidity, 
robustness, and an active retinal vessel segmentation method 
based on supervised classification using a collective 
classifier of boosted and bagged decision trees. 

III. PROPOSED METHOD 


Usually, the suggested method contains three 
phases (Fig. 2). The first phase is the data acquisition 
procedure, which purpose is to extract the features for cell 
nuclei in lung needle biopsy images. Later this process and 
the technique under goes over the rest of the two training 
and testing phases. In the training phase, the new idea of 
dictionary in the pattern. Recognition/computer vision, 
where dictionary resources a collection of elements or words 
or feature vectors. 

A. FRAMEWORK METHOD FOR IMAGE 
PREPROCESSING 


The features extracted from the three modalities 
such as color, texture and shape of every single cell nucleus 
in the data acquisition procedure, which build three original 
sub dictionaries on color, texture and shape by collecting the 
corresponding feature vectors of individual cell nuclei. A 
feature extraction step goal to extract features for distinct 
cell nuclei from three modalities color, texture and shape 
respectively. Exactly, as shown in Table 2, shape based(9), 
color-based (1 1), and texture-based(16) features are mined. 
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Fig.4: Flowchart of the data acquisition -procedure in 
our method, including three sequential steps: image 
capture, image preprocessing, and feature extraction. 
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Fig.5: Extraction of features and the label of each testing 
image are determined by voting on the cell -level label. 

Generally, our proposed method contains three 
phases (see Fig. 2). The first phase is the data acquisition 
procedure, which aims to extract the features for cell nuclei 
in lung needle biopsy images. After this phase, our method 
goes through the rest two ones: training and testing phases. 
In the training phase, we introduce the concept of dictionary 
[9] in pattern recognition/computer vision, where a 
dictionary means a collection of words/elements/feature 
vectors. 

Traditional SRC belongs to single-modal learning 
approach, which only uses one dictionary. Since the 
information of three with the features extracted from the 
three modalities (shape, color, and texture) of each cell 
nucleus in the data acquisition procedure, we build three 
original sub dictionaries on shape, color, and texture by 
collecting the corresponding feature vectors of individual 
cell nuclei. Note that all the sub dictionaries mentioned in 
the following sections only contain the features coming 
from different individual cell nuclei instead of images. 
Moreover for each nucleus, its label is initialized as the 
same label of the image it belongs to. The sub dictionaries 
obtained in this way usually contain several similar samples 
coming from different classes, which might be harmful for 
classification. To learn discriminative sub dictionaries, we 
propose a genetic algorithm-based multimodal dictionary 
learning algorithm, which selects the topmost discriminative 
training cell nuclei, and encourages large disagreement 
among different sub dictionaries. In the testing phase, for a 
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new coming image, a hierarchical fusion strategy is adopted, 
including cell-level fusion and image level fusion, 
respectively. In the following sections, we will discuss the 
technical details of each phase in our method. 

B. DATA ACQUISITION PROCEDURE 

With the goal of capturing and preprocessing the lung 
needle biopsy images, the data acquisition procedure, whose 
main pipeline can be referred to Fig. 3, takes the following 
sequential 
Steps: 

1) An image capturing step aims to capture the images 
from the needle biopsy specimens by using an electronic 
microscopy and a digital camera. 

2) An image preprocessing step aims to obtain the 

individual cell nuclei from the captured images by 
segmenting the cell nuclei from the background (i.e., cell 
sap). The image preprocessing step is with the following sub 
steps: smoothing the images with Gaussian kernel, 

segmenting the images using Otsu’s algorithm, and labeling 
the connected cell nuclei regions. The reason of adopting 
Otsu’s algorithm is that the contrast between the cell nuclei 
region and background is large enough to be easily 
separated (see sample images in Fig. 1). Basically, the 
segmentation results can meet clinical requirements 
according to the pathologist’s suggestions. 

3) A feature extraction step aims to extract features 
for individual cell nuclei from three modalities (shape, 
color, and texture), respectively. Specifically, as shown in 
Table I [1], nine table extracted features for each cell 
nucleus region from three modalities: shape, color, and 
texture shape (9) color (11) texture (16) height R, G, B 
energy (4) width H, S, I entropy (4) [2] circumference gray 
mean contrast (4) [2] area gray variance divergence (4) 
circularity feature gray elongation IOD Fourier descriptor 
(3)[2] central moment shape-based, 11 color-based, and 16 
texture-based features are extracted. For the Fourier 
descriptor, the number “3” in the bracket means that we only 
use the second, third, and fourth coefficients. The first 
coefficient is the mean value of boundary coordinates, 
which is usually considered as useless in feature 
representation. In the texture-based features, the number “4” 
in a bracket means that we calculate the four directions, (0°, 
45°, 90° , 135°) for the four features (energy, entropy, 
contrast, and divergence) [2], so totally 16 texture-based 
features are extracted [2]. These features have demonstrated 
their robustness and effectiveness in several medical image 
analysis applications. 

IV. CONCLUSION 

The novel technique is suggested MSRC for categorizing 
the lung needle biopsy images. MSRC goal is to raise the 
classification performance, which exactly for the images of 
various cancerous types. The genetic technique goal is to 
select the topmost discriminative samples for every single 
distinct modality as well as to assurance the huge diversity 
among different features. From the perception of 
experimental practice, Misclassifying a cancerous image as 
a standard or normal one will be considerably more serious 
than misclassifying a standard or normal image as a 
cancerous one, Forthcoming work will examine how to 


implement the technique on the image set through various 
class relations, i.e., considering the ratio of malignant nuclei 
in every image. Similarly the multimodal data is broadly 
obtainable in medical image exploration due to the 
numerous data acquisition methods. 
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